Anomaly Detection in Log Files Using Selected Natural Language Processing Methods

In this article, we address the problem of detecting anomalies in system log files. Computer systems generate huge numbers events, which are noted event While most them report normal actions, an unusual entry may inform about a failure or malware infection. A human operator easily miss such entry; therefore, anomaly detection methods used for purpose. our work, approach known from natural language processing (NLP) domain, operates on so-called embeddings, that is vector representations words phrases. We describe improved version LogEvent2Vec algorithm, proposed 2020. contrast to original version, propose significant shortening analysis window, both increased accuracy and made further suspicious sequences much easier. experimented with various binary classifiers, as decision trees multilayer perceptrons (MLPs), Blue Gene/L dataset. showed selecting optimal classifier (in case, MLP) short sequence gave very good results. The algorithm yielded best F1-score 0.997, compared 0.886 algorithm.